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Foreword 


This  final  report  covers  the  work  performed  under  Contract 
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Division,  Fort  Worth,  Texas  and  Vanderbilt  University,  Nashville, 
Tennessee.  The  program  was  managed  by  Dr.  B,  G.  W.  Yee  of 
General  Dynamics,  with  Dr.  J.  C.  Couchman  and  Dr,  F.  H,  Chang 
serving  as  principal  investigators.  Valuable  contributions  were 
made  to  the  development  of  the  statistical  analysis  by  Dr,  G.  H. 
Lemon  and  computer  programming  by  J,  S,  Kunselman  and  T.  Walker. 
Dr,  P,  F.  Packman  of  Vanderbilt  University  served  as  associate 
program  manager.  The  program  was  under  the  technical  direction 
of  Mr.  S.  J.  Klima,  NASA  Lewis  Research  Center,  Cleveland,  Ohio. 

Participants  in  this  program  are  indebted  to  many  people 
in  the  NDE  community  for  furnishing  data  to  this  program  and 
for  providing  consultation. 
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SUMMARY 


The  overall  objective  of  this  program  is  to  assess 
available  nondestructive  testing  data  for  the  determination 
of  the  sensitivity  and  reliability  of  state-of-the-art 
production  NDE  methods  for  flaw  detection  on  metallic  ma¬ 
terials.  This  program  was  separated  into  four  different 
tasks.  They  Were: 

Task  I  Acquisition  of  Information 

Task  II  Screening  and  Separation  of 

Data  by  NDE  Method  and  Ma¬ 
terial 

Task  III  Statistical  Determination  of 

NDE  Reliability 

Task  IV  Reporting 

\ 

Twenty  sets  of  relevant  NDE  reliability  data  have  been 
identified,  collected,  compiled,  and  categorized.  Three 
relevant  on-going  programs  have  also  been  identified.  A 
criterion  for  the  selection  of  data  for  statistical  analysis 
considerations  has  been  formulated.  A  model  to  grade  the 
quality  and  validity  of  the  data  sets  has  been  developed. 

Data  input  formats,  which  record  the  pertinent  parameters 
of  the  defect /specimen  and  inspection  procedures,  have  been 
formulated  for  each  NDE  method.  A  comprehensive  computer 
program  has  been  written  to  calculate  the  probability  of 
flaw  detection  at  several  confidence  levels  by  the  binomial 
distribution.  This  program  also  selects  the  desired  data 
sots  for  pooling,  and  tests  the  statistical  pooling  criteria 
before  calculating  the  composite  detection  reliability. 
Probability  of  detection  curves  at  95  and  50  percent  con¬ 
fidence  levels  have  been  plotted  by  NDE  technique  and  ma¬ 
terial  type  for  individual  sets  of  data  as  well  as  for  merged 
data. 
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I.  INTRODUCTION 


In  order  to  apply  linear-elastic  fracture  mechanics  to 
structural  design,  NDE  has  to  show  at  a  high  level  of  con¬ 
fidence  that  no  flaw  larger  than  a  specific  size  exists  in 
the  structure.  To  establish  the  minimum  detectable  flaw 
size,  many  companies  and  organizations  have  conducted  NDE 
demonstration  programs.  Most  of  these  demonstration  programs 
have  been  conducted  in  the  production  and  field-service  environ 
ment,  but  some  have  been  conducted  in  the  laboratory  environ¬ 
ment  . 

The  results  obtained  from  demonstration  programs  are 
lacking  in  universal  agreement.  This  lack  of  agreement  is 
not  surprising  because  each  company  or  organization  may  use 
a  different  NDE  procedure,  different  personnel,  different 
procedures  and  parameters  to  generate  the  test  flaws,  different 
flaws  and  material  types,  and  even  different  statistical  ana¬ 
lysis  procedures.  There  appears  to  be  a  need  to  (1)  collect 
much  of  the  available  NDE  reliability  data,  (2)  closely  examine 
all  the  parameters  that  could  affect  the  detection  reliability, 
(3)  compare  the  parameters  used  by  each  organization  to  obtain 
the  data,  and  (4)  attempt  to  identify  the  parameters  that  most 
likely  cause  observable  differences  in  detection  reliability. 

It  appears  worthwhile  to  obtain  a  composite  detection  relia¬ 
bility  for  each  NDE  method,  material  t5rpe,  and  flaw  type  by 
pooling  data  obtained  from  several  sources.  At  the  same  time, 
the  merits  and  shortcomings  of  several  statistical  analysis 
procedures  should  be  carefully  examined  and  the  procedure  most: 
suitable  for  the  analysis  of  NDE  reliability  data  should  be 
selected.  Any  needs  for  improved  methods  should  be  identified. 

The  program  reported  on  in  this  docimient  was  intended  to 
collect  all  available  NDE  data,  screen  and  separate  the  data  by 
NDE  method  and  material,  perform  statistical  analyses,  and 
evaluate  the  state-of-the-art  in  NDE  reliability. 
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II.  ACQUISITION  OF  INFORMATION 


The  acquisition  of  NDE  reliability  related  data,  identifi¬ 
cation  of  on-going  programs,  and  preparation  of  a  bibliography 
of  the  acquired  data  is  discussed  in  this  section. 

2.1  Acquisition  of  NDE  Reliability  Related  Data 

Twenty- three  sets  of  potentially  useful  data  have  been 
identified  during  this  program  and  twenty  sets  were  acquired. 
Three  sets  of  data  involve  on-going  programs  and  the  data  were 
not  made  available  for  inclusion  in  this  study.  Of  the  twenty 
sets  of  data  received,  only  seven  sets  were  statistically 
analyzed.  The  thirteen  sets  of  data  that  were  not  statisti¬ 
cally  analyzed  were  rejected  because  they  did  not  satisfy  the 
selection  criterion  discussed  in  Section  III  of  this  report  or 
the  owner  refused  to  permit  the  data  to  be  used. 

2.2  Bibliography 

The  twenty  sets  of  NDE  reliability  related  data  that 
have  been  acquired  are  listed  below.  Some  of  the  references 
are  private  data  in  which  case  only  company  or  committee  report 
numbers  are  available.  Several  references  to  government  funded 
programs,  which  have  not  been  published  can  only  be  identified 
by  sponsoring  agencies  and  the  name  of  an  individual  at  the 
company  where  the  work  was  conducted.  Copies  of  these  data 
can  be  obtained  by  either  contacting  an  individual  within  the 
sponsoring  agency  or  an  individual  associated  with  the  company. 
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2.3  On - Go ing  P  rog  rams 

There  are  several  on-going  programs  that  are  currently 
known.  Of  these,  only  three  are  government  funded  and  their 
results  will  be  available  to  the  public  for  analysis.  The 
data  from  the  privately  funded  programs  may  never  be  made 
available  for  public  analysis. 

The  three  government  funded  on-going  programs  are: 

1.  Crack  detection  reliability  on  welded  plates  and 
structures,  Martin  Marietta,  Ward  Rummel,  sponsored 
by  NASA/Johnson  Space  Center,  NAS  9-13578. 

2.  Crack  detection  reliability  on  actual  aircraft 
structures  at  the  depot  level,  Lockheed,  GA., 

W.  Lewis,  sponsored  by  Kelly  Air  Force  Base. 

3.  Crack  detection  reliability  on  bolt  holes  on  F-111 
fatigue  tested  structures.  General  Djmamics,  Fort  Worth 
Division,  B,  G,  W,  Yee,  Sponsored  by  SMALC. 
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III.  SCREENING  AND  SEPARATION  OF  DATA  BY 
NDE  METHOD  AND  MATERIAL 


This  section  describes  the  development  of  data  selection 
criteria,  separation  and  categorization  of  data,  development 
of  a  model  to  grade  quality  of  the  data,  and  the  development 
of  a  data  input  format  for  each  NDE  method. 

3,1  Criteria  for  Selection  of  Data  for  Statistical  Analysis 

All  of  the  reliability  related  NDE  data  are  not  necessarily 
suitable  for  statistical  analysis.  Some  are  lacking  in  the 
documentation  of  certain  key  pertinent  parameters,  such  as  the 
defect  dimension,  defect  type,  NDE  method,  etc.  Statistical 
analysis  of  data  when  the  key  pertinent  parameters  are  not 
documented  would  be  marginal  in  value.  A  data  selection 
criteria  is  needed  to  screen  the  data  and  prejudge  the  suitability 
of  the  data  for  statistical  analysis.  Such  a  criteria  is 
necessarily  subjective  because  it  involves  human  judgment  of 
data  value  or  usefulness.  It  is  felt  that  such  a  subjective 
criteria  will  still  be  useful  to  screen  out  data  having  margi¬ 
nal  statistical  value  and  to  eliminate  lost  time  in  processing 
the  data. 

To  be  eligible  for  statistical  analysis,  a  set  of  data 
must  satisfy  the  following  conditions: 

a)  An  NDE  procedure  or  specification  must  accompany  the 
data  which  clearly  describes  the  equipment  and  the 
parameters  used  so  that  the  data  may  be  reproduced 
in  other  facilities  (assuming  the  same  equipment  or 
its  equivalent  is  used) . 

b)  The  defect  dimensions  and  specimen  geometry  must  be 
well  dociraiented  so  that  data  may  be  statistically 
analyzed  and  compared  to  defect  detection  in  the 
proper  defect  size  range.  When  artificial  methods 
of  defect  fabrication  are  used,  at  least  ten  percent 
(10%)  of  all  defects  in  a  given  set  must  be  destruc¬ 
tively  tested  to  obtain  the  defect  dimensions.  For 
methods  that  are  used  to  produce  multiple  defects  in 
a  specimen  or  methods  that  are  questionable  for 
producing  controllable  defect  dimensions,  at  least 
fifty  percent  (50%)  of  all  defects  in  a  given  set 
must  be  destructively  tested  to  verify  the  defect 
dimensions. 


6 


The  model  that  was  developed  to  grade  the  quality  of  the 
data  was  very  subjective.  It  consisted  of  assigning  weighting 
factors  and  summing  up  the  weighting  factors  to  obtain  an  overall 
factor  that  is  an  index  of  data  quality.  The  model  is  in¬ 
cluded  in  Appendix  A. 

Data  pooling  tests  that  were  coded  in  the  NASA  data  pro¬ 
cessing  code  (reproduced  in  Appendix  B)  provided  a  means  for 
isolating  input  errors  and  identifying  any  data  set  that  may 
be  suspect.  The  tests  are  derived  from  binomial  statistics 
equations  which  are  developed  in  the  next  section. 

The  computer  code  developed  for  processing  the  NDE  relia¬ 
bility  data  is  described  in  Appendix  -C.  Included  is  a  complete 
description  of  data  input  formats  and  parameter  keys. 

3.2  Separation  and  Categorization  of  Data  Sets 

The  twenty  sets  of  data  listed  in  the  Bibliography  in 
Subsection  2.2  can  be  separated  into  three  categories.  Table 
3-1  describes  the  data  separated  into  the  three  categories  and 
the  status  of  these  data  sets.  The  first  category  is  the  data 
that  appear  to  satisfy  the  criteria  discussed  in  Subsection 
3.1,  and  they  will  be  considered  for  statistical  analysis. 

There  are  seven  (7)  sets  of  data  in  this  category.  The  second 
category  is  data  that  probably  could  be  used  if  permission  to 
use  the  data  is  granted  by  the  rightful  owner  of  the  data.  There 
are  two  sets  of  data  in  this  category.  The  third  category  is  the 
data  that  are  either  lacking  in  the  inspection  procedure  docu¬ 
mentation  or  defect  dimension  documentation.  There  are  eleven  sets 
of  data  in  this  category.  The  data  sets  in  each  of  the  three 
categories  are  presented  in  Table  3-1, 

A  large  majority  of  the  data  were  obtained  on  thin  flat 
plates  which  contain  fatigue  cracks  or  weld  defects.  There  are 
few  sets  of  data  that  were  obtained  with  relatively  complex 
shaped  specimens  such  as  a  T,  I,  or  H  shape.  In  order  to  gain 
a  better  understanding  of  the  availability  of  data  on  material 
type,  defect  type,  and  specimen  complexity,  a  table  (Table  3-2) 
is  constructed  to  categorize  the  data  sets  according  to  test 
specimen  complexity.  Within  each  data  set,  a  brief  description 
of  the  material  t5q>e,  NDE  methods,  and  defect  t3!pe  is  presented. 

The  materials  and  shapes  for  which  valid  data  were  obtained 
and  statistical  analysis  results  are  computed  are  tabulated 
later  in  Section  V  (RESULTS), 
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Table  3-1  STATUS  AND  CATEGORY  OF  DATA 
CONSIDERED  FOR  NDE  RELIABILITY  ASSESSMENT 
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Table  3-2  Data  Grouping  According  to  Specimen 
Thickness  and  Complexity 
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IV  STATISTICAL  DETERMINATION  OF  NDE  RELIABILITY 


This  section  describes  the  binomial  statistical  method, 
cumulative  schemes,  statistical  pooling  procedure,  and  digital 
computer  code  for  computing  NDE  reliability. 


4.1  Introduction 

There  are  four  possible  outcomes  from  any  nondestructive 
inspection  of  an  item:  (1)  detection  of  a  defect  that  is 
present,  (2)  non-detection  of  a  defect  that  is  present, 

(3)  detection  of  a  defect  that  is  not  present  (false  indication), 
and  (4)  non-detection  of  a  defect  that  is  not  present.  Because 
of  these  four  possible  outcomes,  any  single  inspection  may  be 
called  a  quadrinomial  event.  Although  it  is  recognized  that 
false  indications  of  defects  and  true  indications  of  non¬ 
defective  items  (cases  3  and  4)  are  of  practical  significance 
to  both  the  manufacturer  and  the  customer,  it  is  beyond  the 
scope  of  this  investigation  to  develop  a  straightforward 
statistical  method  for  handling  the  quadrinomial  event. 

Preliminary  indications  show  that  most  NDE  reliability 
investigations  have  neglected  to  report  information  concerning 
either  false  indications  of  defects,  or  true  indications  that 
specimens  contained  no  intentionally  induced  flaws.  However, 
the  data  input  format  discussed  in  the  previous  section  provides 
for  storage  of  information  concerning  false  indications  for 
future  use  when  more  of  these  data  become  available. 

Cases  (1)  and  (2)  involve  either  a  detection  or  non¬ 
detection  of  a  defect  that  is  known  to  exist.  This  event  can 
best  be  described  statistically  by  applying  the  binomial  distri¬ 
bution.  The  Normal,  Chi-square,  and  Poisson  distributions  are 
sometimes  used  as  approximations  to  the  binomial.  Their 
applicability  to  the  problem  of  NDE  reliability  is  to  be 
considered  in  the  later  sections  of  this  report. 

4.2  Application  of  Binomial  Distribution 

An  event  that  has  only  two  possible  outcomes  is  referred 
to  as  a  binomial  event.  Suppose,  for  example,  an  experiment 
in  NDE  is  performed  where  N  specimens,  all  containing  identical 
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flaws,  are  routed  through  an  ultrasonic  inspection  system. 
Suppose  further,  that  the  system  capability  does  not  change 
throughout  the  entire  inspection  process,  i.e.,  each  specimen 
is  evaluated  independently  of  the  others.  Let  p  equal  the 
true, (but  as  yet  unknown)  probability  of  detecting  each  flaw 
and  q=l-p  be  the  probability  of  missing  each  flaw.  Assuming 
p  remains  the  same  for  all  specimens,  the  random  variable  X 
can  be  defined  as  being  the  number  of  flaws  that  are  detected 

in  any  given  experiment,  X,  then,  is  referred  to  as  a  binomial 
random  variable  with  parameters  N  and  p.  Its  possible  values 
are  0,  1,  2,  ...,  N.  Equivalently,  it  can  be  said  that  X  has 
a  binomial  distribution.  The  probability  of  obtaining  any  one 
of  the  N+1  possible  values  of  X  from  such  an  experiment  is 
described  by  the  following  equation: 


P(x^)  p 


N-n 

q 


j 


where 


Nl 

nI(N-n)I 


n=0,l, . . .  ,N 


(1) 


The  sum  of  all  the  possible  values  for  equation  (1)  is  equal 
to  unity  and  can  be  written  as  follows: 


(p+q)^ 


N 

z 


n=o 


n 

P 


N-n 

q 


1 


(2) 


The  probability  of  detecting  n  or  more  flaws  can  be  found  by 
summing  equation  (1)  over  all  the  values  of  X  for  which 
X  >  n .  Thus , 


N 

P(X^n)  = 

3.=n 


i  N- 

p  q 


(3) 


4,2.1  Confidence  Interval  Estimates  of  the  True  Probability 
of  Detection 

The  objective  is  to  estimate  the  true  proportion  of 
defects  of  a  particular  type  and  size  that  can  be  detected  by 
a  given  NDE  method.  The  best  single  estimate,  p,  of  the  true 
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detectable  proportion  is  the  number  of  flaws  detected  divided 
by  the  total  number  of  flaws  present: 


P  * 


n 

N 


(4) 


If  the  binomial  experiment  (N,  p)  is  repeated  an 
infinite  number  of  times  and  if  is  computed  each  time, 
then  the  average  of  all  the  p^s  will  be  equal  to  po  So  p 
is  an  unbiased  estimator  of  the  true  probability  of  crack 
detection. 


Since  the  probability  is  small  that  p  will  exactly  equal 
p  on  any  specific  replication  of  the  experiment,  an  interval 
estimator  that  will  contain  p  most  of  the  time  is  considered 
useful,  A  lower  one-sided  confidence  interval  can  be  used  to 
estimate  a  lower  bound  on  the  true  probability  of  crack  de¬ 
tection.  The  lower  bound,  p^^,  for  n  detections  of  N  cracks 
is  computed  as  follows: 


Solve  S(i)  Pj^  (l-p^)^*’" 

i  "  n 


for  pj^,  where 

«=  1  -  G  (6) 

The  following  interpretation  can  be  given  for  a 
lOOG  percent  lower  binomial  confidence  interval.  If  the 
binomial  experiment  (N,  nj^,  p)  is  repeated  many  times 
(theoretically  infinite),  and  if  pj^^^  is  computed  each  time, 
then  lOOG  percent  of  these  lower  confidence  intervals  will 
be  equal  to  or  lower  than  p.  Thus,  there  is  lOOG  percent 
"confidence"  that  the  lower  bound,  pj^,  computed  for  any 
specific  binomial  experiment  will  be  less  than  or  equal  to 
the  true  probability  of  crack  detection.  The  choice  of  G 
is  arbitrary  and  depends  on  how  sure  one  needs  to  be  that  the 
true  probability  of  detection  is  in  the  interval  from  p^  to  1, 
(the  larger  the  value-  of  G,  the  smaller  the  calculated  lower 
bound  pj  will  be) , 

*  is  the  number  of  successes  in  the  Kth  replication  of  a 
binomial  experiment  consisting  of  N  measurements. 


15 


4,2.2  Sample  Size  Detennination 

The  objective  is  to  detennine  the  sample  size  required 
to  estimate  the  lower  confidence  limit  and  confidence  level. 

This  can  be  accomplished  by  utilizing  equation  (^).  By 
specifying  the  confidence  level,  G,  and  the  lower  confidence 
limit  p, ,  a  set  of  values  (which  must  be  integers)  can  be 
computed  for  N  and  n.  Each  combination  of  N  and  n  in  this 
set  indicates  the  number  of  inspections  and  the  number  of 
detections  required  to  achieve  the  specified  probability  of 
detection  (POD)  at  the  stated  confidence  level.  For  example,  if 
G  and  p^^  are  chosen  to  be  0.95  and  0.9  respectively,  equation 
(  5)  becomes 


0.95  =  1  -  ^  (i)  (0. 9)^(0. 1)^"^. 

x^n  W 


(7) 


One  of  the  combinations  of  N  and  n  is  29  and  29  respectively. 
This  represents  the  smallest  sample  size  that  can  be  utilized 
to  meet  the  minimum  specified  values  for  G  and  p^.  The  next 
smallest  sample  size  is  N=46,  In  this  case  n  must  equal  at 
least  45  to  achieve  907o  probability  of  detection  at  957o 
confidence  level.  The  higher  the  reliability  requirements, 
of  course,  the  larger  the  sample  size  required. 


Equation  (5)  can  also  be  used  to  calculate  the  number 
of  added  NDE  tests  required  to  upgrade  an  existing  batch  of 
data  in  the  hope  of  achieving  higher  reliability  estimates. 
Equation  (5)  takes  on  the  form 


1-G 


6-€ 


tl 


(1-Pl) 


N+d-i 


where  6  is  the  required  number  of  additional  tests  and 
€  is  the  maximum  number  of  additional  misses  (nondetections)  .. 
For  example,  if  an  experiment  consisting  of  29  inspections 
and  28  detections  was  performed,  the  reliability  (equation  5) 
is  90%  probability  of  detection  at  80%  confidence  level.  If 
a  95%  confidence  level  is  desired,  the  additional  data  require¬ 
ments  are  indicated  by  equation  (8).  Thus,  6-17  with  £  “  0 
represents  the  minimum  added  sample  required  to  upgrade  the 
existing  data. 
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4.3  Comparison  of  Alternative  Statistical  Procedures 

Table  4-1  lists  the  probability  of  detection  at  the  95% 
confidence  level  for  four  commonly  used  probability  analysis 
methods,  (Binomial,  Normal,  Poisson,  and  Chi-Square).  The 
comparison  is  on  the  basis  of  30  trials  with  10,  15,  20,  25, 

29  and  30  detections.  The  Binomial  gives  the  exact  results. 

The  Normal  approximation  gives  confidence  values  greater  than 
the  Binomial  for  large  successes  (30/30),  and  very  nearly 
equal  to  the  Binomial  for  intermediate  successes  (15  or  20/30). 

The  Poisson  approximation  gives  values  less  than  the  Binomial 
method.  It  is  a  good  approximation  only  for  cases  with  large 
number  of  trials  with  small  number  of  successes.  For  30  trials, 
the  Chi-Square  method  gives  values  less  than  the  Binomial  for 

10  to  30  successes.  Like  the  Poisson,  the  Chi-Square  values 
approach  those  of  the  Binomial  for  less  than  10  successes  in 

30  trials. 

However,  a  closer  approximation  to  the  Binomial  for 
both  the  Poisson  and  Chi-Square  can  be  achieved  by  using  a 
conditional  approach.  By  first  calculating  the  upper  one-sided 
confidence  limit  for  the  probability  of  missing  a  flaw  (9^),  the 
lower  one-sided  confidence  limit  for  P^  =  1  -  9^^  can  then  be 
approximated  when  1  -  n^n.  By  using  this  conditional  approach,  a  new 
set  of  values  is  calculated  and  included  in  Table  4-1  under  the 
conditional  approach  column.  These  new  values  give  a  much  better 
approximation  to  the  Binomial  when  the  number  of  successes  is  high. 

Since  the  Poisson  and  Chi-Square  approximations  yield  almost 
the  same  values,  only  one  set  of  values  was  calculated  under 
the  new  conditional  approach. 

For  larger  or  smaller  number  of  trials,  the  above  com¬ 
parisons  is  not  necessarily  true  but  the  comparison  generally 
shows  the  importance  of  analyzing  binomial  measurements  with 
binomial  statistics. 
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4.4  Data  Cumulation  Methods 


The  calculated  value  for  the  lower  confidence  limit 
(probability  of  detection,  p^,  at  some  selected  confidence 
level,  CL)  is  influenced  by  the  total  number  of  measurements 
(sample  size).  In  order  achieve  a  high  POD  at  a  high  CL, 
such  as  90%  at  95%  confidence  level,  a  minimum  of  29  measure¬ 
ments  have  to  be  made  without  a  miss  for  a  given  flaw  size. 
Because  of  the  high  costs  involved,  it  is  generally  not 
economical  to  make  29  or  more  measurements  for  each  flaw 
size  for  the  entire  range  of  flaw  sizes  of  interest.  At  the 
same  time,  in  the  inspection  of  actual  structural  components, 
it  is  unlikely  to  have  29  or  more  measurements  at  any  specific 
flaw  size.  As  a  result,  size  interval  grouping  has  been  used 
in  order  to  obtain  a  sufficient  number  of  measurements  to 
compute  a  high  p]^  at  a  high  CL  and  to  smooth  over  the  flaw 
sizes  that  have  no  measurements.  A  cumulation  plan  permits 
the  accumulation  of  data  over  a  range  of  flaw  sizes  for 
computing  a  p]^  which  is  representative  of  that  range. 

Several  cumulation  procedures  were  considered.  These 
procedures  include  (1)  the  range  interval  "RI",  (2)  the 
overlapping  60  points  ‘'OSP"  and  (3)  the  procedure  developed 
under  this  contract  which  will  be  called  the  optimized 
probability  method  "0PM. ”  Other  procedures  evaluated  are 
reported  in  Reference  1.  For  the  same  set  of  data,  the  computed 
PI  can  be  considerably  different  depending  on  which  of  these 
cumulative  procedures  is  used.  Generally,  NDE  reliability  is 
demonstrated  by  inspecting  specimens  containing  flaws  distributed 
uniformly  over  a  wide  flaw  size  range.  The  smallest  flaws  should 
be  virtually  nondetectable  and  the  largest  flaws  should  be  100% 
detectable.  First,  the  raw  data  set  is  arranged  in  order  of 
increasing  flaw  size  with  the  appropriate  outcome  indicated  for 
each  measurement.  The  various  cumulative  procedures  are  applied 
as  follows: 

(a)  Range  interval  method.  The  data  are  separated  into  groups 
of  equal  flaw  size  increments.  The  probability  of  detection  at 
the  one  sided  lower  confidence  limit  is  computed  for  each  group 
separately  and  plotted  as  a  histogram  bar.  A  conservative  p^^ 
curve  can  be  obtained  by  connecting  the  upper  right  hand  corner 
of  the  bars.  This  procedure  is  most  appropriate  for  computing 
probabilities  of  detection  and  confidence  limits  when  large 
numbers  of  data  are  available  so  that  all  histogram  bars  repre¬ 
sent  large  data  samples.  It  is  important  to  note  that  sample 
size  (N)  may  vary  widely  betweeti  intervals. 
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(b)  Overlapping  sixty  point  method  (N  =  60).  One  begins  by 
combining  detection  results  for  the  largest  sixty  crack  sizes. 
The  POD  is  calculated  for  this  interval  and  plotted  at  the 
largest  flaw  size  within  the  size  range  spanned.  The  next 
data  increment  is  obtained  by  starting  at  the  median  flaw 
size  of  the  first  interval  and  combining  the  data  for  the 
next  smaller  60  cracks.  The  POD  of  the  second  set  is  plotted 
at  its  largest  flaw  size  and  the  process  is  repeated  until  all 
data  have  been  combined.  Each  interval  overlaps  its  adjacent 
interval  by  30  data  points.  Note  that  the  sample  size  is  the 
same  for  each  interval,  but  the  breadth  of  the  interval  can 
vary  widely. 

(c)  Optimized  Probability  Method.  The  ordered  NDE  data  are 
grouped  into  J  (a  computer  input  number)  intervals  of 
successively  increasing  size  range.  The  POD  of  the  largest 
size  range  is  computed  at  some  desired  confidence  level.  The 
next  smaller  size  range  data  are  combined  with  the  first  and 
the  POD  of  the  second  grouping  is  computed.  This  process  is 
continued  until  J  probabilities  of  detection  are  computed, 

and  the  largest  value  of  POD  obtained  is  plotted  at  the  largest 
flaw  size  contained  in  the  corresponding  composite  grouping. 

The  largest  flaw  size  interval  is  removed  from  consideration 
and  this  procedure  is  repeated  starting  from  the  next  to 
largest  flaw  size  grouping.  The  pattern  is  repeated  until  J 
probability  of  detections  can  be  plotted.  Note  that  if  J  is 
sufficiently  large,  the  POD  curve  produced  by  this  method  will 
be  the  upper  envelope  of  either  the  RI  or  the  OSP  method. 

The  advantage  to  this  method  is  that  the  sample  size  (N)  is 
maximized,  and  results  in  the  maximum  possible  value  of  pj^  for 
the  available  data.  A  basic  assumption  is  that  the  larger  the 
flaw,  the  more  detectable  it  is. 
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V.  RESULTS 


Of  the  twenty  sets  of  data  collected,  only  seven  sets 
were  statistically  analyzed.  These  seven  sets  appeared  to 
meet  the  criteria  stated  in  Section  III  of  this  report.  Each 
of  these  seven  sets,  in  turn,  was  subdivided  into  subsets  and 
analyzed  according  to  the  NDE  method  and  material  types. 
Appendix  D  contains  a  computer  listing  of  NDE  data  and  de¬ 
tection  reliability  results  that  were  analyzed  during  this 
program.  Table  5-1  summarizes  the  results  of  the  statistical 
analysis.  There  is  a  total  of  one  hundred  and  twelve  subsets 
and  the  subset  number  is  given  in  the  first  column  of  Table  5-1. 
The  second  column  gives  data  source.  The  third  and  fourth 
columns  identify  the  material  and  defect  type  respectively. 

The  fifth  and  sixth  columns  identify  the  specimen  geometry 
and  NDE  method  respectively.  The  seventh  column  lists  some  of 
the  pertinent  parameters,  and  the  eighth  column  gives  the  crack 
length  at  which  90%  probability  of  detection  at  95%  confidence 
level  P0D90(CL95)  was  first  achieved.  The  shortest  crack 
length  that  reached  POD90(CL95)  by  either  the  0PM  or  OSP  scheme 
was  used  and  is  herein  referred  to  as  the  threshold  level.  In 
many  instances,  the  POD(CL95)  became  smaller  than  90%  at 
crack  lengths  above  the  threshold  level.  This  is  particularly 
true  for  the  OSP  scheme  (see  Appendix  D) .  For  more  details 
on  the  NDE  parameters  and  inspection  procedures  that  were  used 
to  acquire  each  set  or  subset  of  data,  one  has  to  refer  to  the 
original  data  reference. 

The  first  set  of  data  analyzed  is  from  Martin  Marietta 
(Contract  NAS  9  12276).  Four  subsets,  one  by  ultrasonic 
surface  wave,  one  by  penetrant,  one  by  eddy  current,  and  one 
by  the  X-ray  method,  were  available  before  the  specimens  were 
chemically  etched  and  another  four  subsets  were  available 
after  chemically  etching  the  test  specimens. 

The  second  and  third  sets  of  data  analyzed  are  from 
Rockwell  International-Space  Division  (Contract  NAS  9-14000) 
and  General  D3mamics,  Convair  Aerospace  Division  (Contract 
NAS  9-12326).  This  set  of  data  includes  data  from  Contract 
NAS  9-12276  with  the  specimens  in  the  "after-etched"  condition. 
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There  are  five  subsets  of  data  obtained  with  the  ultrasonic 
method,  each  subset  for  a  different  inspector,  three  inspectors 
for  Contract  NAS  9-12276,  one  for  NAS  9-14000,  and  one  for 
NAS  9-12326.  There  are  seven  subsets  of  data  obtained  with 
the  liquid  penetrant  method,  each  subset  by  a  different 
inspector,  three  inspectors  for  NAS  9-12276,  three  for  NAS  9- 
14000,  and  one  for  NAS  9-12326,  There  are  five  subsets  of 
data  obtained  with  the  eddy  current  method,  each  subset  by  a 
different  inspector,  three  inspectors  for  NAS  9-12276,  one  for 
NAS  9-14000,  and  one  for  NAS  9-12326.  There  are  seven  subsets 
of  data  obtained  with  the  X-ray  method,  each  subset  by  a 
different  inspector:  three  inspectors  for  NAS  9-12276,  three 
for  NAS  9-14000,  and  one  for  NAS  9-12326.  Subsets  33,  34  and 
35  were  obtained  by  merging  the  data  of  five  ultrasonic 
inspectors,  seven  penetrant  inspectors,  and  five  eddy  current 
inspectors.  The  X-ray  data  were  not  merged  because  the  results 
are  not  worthy  of  further  consideration. 

The  fourth  set  of  data  analyzed  is  from  Rockwell 
International  B-1  Division,  B-1  NDI  Demonstration  Program. 
There  were  twelve  subsets  of  data  in  this  set.  The  fifth 
set  of  data  analyzed  is  from  Lockheed,  Georgia,  AFML  Report 
No.  TR-68-32.  There  are  eight  subsets  in  this  data  set.  The 
sixth  set  of  data  analyzed  is  from  Boeing  Company  of  Wichita, 
Kansas .  This  set  of  data  was  obtained  from  eddy  current 
inspection  of  bolt  holes,  after  pulling  off  the  bolts,  from 
actual  aircraft  parts. 

The  seventh  set  of  data  analyzed  is  from  the  Boeing 
Commercial  Airplane  Company,  AFML  Report  No.  TR-74-241,  There 
are  fifty- three  subsets  in  this  set  of  data. 

Data  subset  number  48  was  obtained  by  merging  the  data 
from  Martin  Marietta  (NAS  9-12276),  Rockwell  Int. -Space 
Division  (NAS  9-14000),  General  D3mamics'  Convair  Division 
(NAS  9  12326),  and  Roc^ell  Int, -B-1  Division  for  aluminum 
for  the  liquid  penetrant  method. 
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All  the  data  described  in  this  report  were  taken  from 
a  total  of  nine  alloys  of  aluminum,  titanium,  and  steel. 
These  alloys  are  described  in  Table  5-2,  The  predominant 
amount  of  data  were  taken  with  the  four  aluminum  alloys. 


Table  5-2  Alloys  with  Valid  Data 


Aluminum 

Steel 

Titanium 

2219-T87 

PH13-8MO 

6A1-4V 

7075-T6511 

4330V 

2024-T6 

PH17-4 

7178-T651 

4340M 
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VI.  DISCUSSION  OF  RESULTS 


The  tabulated  and  graphical  computer  output  format 
(Appendix  D) ,  is  convenient  for  assessing  the  state-of-the- 
art  in  NDE  reliability.  It  contains: 

(1)  the  binomial  experiment  results  (N,n)  in  each  of  32 

crack  size  intervals  (tabulated  in  units  of  Mils  and 

plotted  in  both  Mils  and  cm.); 

(2)  the  50  percent  lower  confidence  limit  for  each  interval  POD(CL50 

(3)  the  95  percent  lower  confidence  limit  POD(CL95) ;  and 

(4)  the  number  of  new  measurements  required  in  each  interval 
to  demonstrate  a  90  percent  probability  of  detection  at 
a  95  percent  confidence  level  "POD90(CL95)". 

These  data  are  useful  for  comparing  the  effects  of  material, 
inspector,  and  inspection  parameters  on  NDE  reliability  and 
seeing  where  data  deficiencies  lie. 

As  can  be  seen  in  Figure  D-la  the  POD  fluctuates  widely 
throughout  32  ranges  for  this  set  of  data  because  of  the 
variation  in  the  number  of  measurements,  N,  reported  in  each 
range.  Twenty  of  the  ranges  contain  less  than  29  measure¬ 
ments  while  six  ranges  contained  no  measurements  at  all.  The 
number  of  measurements  per  range  can  be  increased  by  broadening 
the  flaw  size  interval  per  range,  but  there  is  still  no 
assurance  that  all  of  the  broader  ranges  will  contain  measure¬ 
ments.  Fluctuation  in  the  probability  of  detection  due  to 
the  variation  in  the  number  of  measurements  in  a  range  is  a 
shortcoming  of  the  range  interval  method. 

Figure  D-lb  shows  the  tabulated  and  graphic  results  ob¬ 
tained  by  using  the  optimized  probability  method  (0PM)  for  the 
set  of  data  in  Figure  D-la.  As  can  be  seen,  the  number  of 
measurements  available  for  each  interval  is  much  larger  than 
for  the  range  scheme.  The  thirty- first  interval  lists  183 
detections  out  of  183  measurements  resulting  in  a  POD  of  987. 
plotted  at  1.18  cm.  The  POD  computed  with  the  0PM  increases 
monotonically  with  increasing  crack  length  and  does  not 
fluctuate  as  in  the  range  scheme.  The  POD  at  95%  CL  reaches 
897o  for  the  first  time  at  .32  cm  for  the  0PM  versus  .358  cm 
for  the  range  scheme. 
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Figure  D-lc  shows  graphic  and  tabulated  results  using 
the  overlapping  60  points  scheme.  In  this  scheme,  the  number 
of  measurements  for  each  POD  calculation  is  constant  (60). 
Fluctuation  in  the  POD  can  be  attributed  primarily  to  human 
operator  and  inspection  process  variations.  The  POD  at  95% 

CL  reaches  89%  for  the  first  time  at  .328  cm,  which  is  inter¬ 
mediate  between  the  0PM  and  the  RI  results. 

6.1  Effects  of  Material,  Source,  and  Inspection  Parameters 

It  is  apparent  from  data  subsets  1  through  8  in  Table 
5-1  that  the  sensitivity  is  increased  for  ultrasonic,  liquid 
penetrant,  and  X-ray  techniques  after  the  test  specimens 
were  chemically  etched.  This  increase  is  reasonable  since 
the  crack  openings  are  enlarged.  The  sensitivity  is  decreased, 
however,  for  eddy  current  method  after  chemically  etching  the 
specimens.  This  effect  is  difficult  to  explain. 

Data  subsets  9  through  32  show  that  the  difference  in 
the  POD90(CL95)  obtained  by  different  inspection  operators 
within  a  company  vary  as  much  as  those  obtained  by  companies 
which  use  different  inspection  parameters  and  procedures. 
Several  observations  can  be  made  with  subsets  9-14: 

(1)  Operators  P  and  S  in  subsets  10  and  13  can  be  considered 
to  be  model  operators.  Not  only  did  they  achieve  a  smaller 
POD90(CL95)  threshold  level  than  other  operators,  they  were 
able  to  maintain  the  POD(CL95)  at  a  relatively  constant 
(and  high)  level  for  crack  lengths  above  the  threshold  level 

(Figures  D- 10  and  D-13>.  (2)  Subsets  10  and  13  show  that  the 

sensitivity  of  the  ultrasonic  method  increases  or  remains 
constant  for  increasing  crack  length,  particularly  for  crack 
lengths  above  the  threshold  level.  (3)  Fluctuations  in  the 
POD(CL95)  for  crack  lengths  above  the  threshold  level  are 
primarily  caused  by  fluctuations  in  operator  efficiency  and 
less  likely  by  the  sensitivity  of  the  NDE  method. 


Subsets  14-20  (Figures  D-14  to  D-20)  show  more  fluctuation 
in  the ; POD(CL95)  due  to  variation  in  operator  efficiency  than 
to  types  of  penetrant  or  procedures  used  by  the  three  different 
companies  that  made  the  measurements.  There  is  a  large  variation 
in  the  threshold  level  of  the  seven  operators.  Three  operators 
demonstrated  threshold  levels  around  .33  cm,  another  three 
demonstrated  the  level  at  .737  cm,  and  the  seventh  one  demonstrated 
it  at  1.35  cm.  One  should  keep  in  mind  that  most  of  these  POD 
(CL95)  curves  are  relatively  flat  and  the  POD(CL95)  is  greatet 
than  80%  from  .254  cm  on  to  the  longer  crack  length. 
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Like  the  ultrasonic  and  penetrant  results,  the  eddy 
current  results  showed  considerable  differences  in  the 
threshold  level  for  the  five  different  operators.  In  fact 
Operator  V  did  not  demonstrate  POD90(CL95).  However,  he 
did  demonstrate  POD88(CL95)  at  a  crack  length  of  .737  cm. 
Operator  T  demonstrated  POD90(CL95)  at  a  crack  length  of 
.665  cm.  However,  he  reached  POD88(CL95)  at  a  crack  length 
of  .399  cm. 

X-ray  is  not  a  good  inspection  technique  for  reliably 
detecting  fatigue  cracks.  Data  sets,  26  to  32  show  that 
only  one  operator  can  achieve  POD90(CL95).  This  forecasts 
that  X-ray  techniques  will  not  be  viable  procedures  for 
detecting  fatigue  cracks.  However,  this  does  not  necessarily 
mean  that  the  technique  should  not  be  used  to  detect  other 
defects  such  as  porosity. 

Merging  data  provides  a  means  for  observing  overall 
detection  reliability  trends.  When  the  data  for  five  ultra¬ 
sonic  operators  were  merged  (subset  No.  33)  and  calculated 
with  the  OSP  scheme  the  POD(CL95)  reached  95%  at  a  crack 
length  of  0.66  cm  (see  Figure  D-33c) .  However,  it  fell  below 
95%  at  higher  crack  length,  and  it  reached  a  minimum  of  79% 
at  a  crack  length  of  0.810  cm.  When  the  results  of  seven 
penetrant  operators  were  merged  (subset  No.  34)  and  calculated 
with  the  OSP  scheme,  the  POD(CL95)  reached  92%  at  a  crack 
length  of  0.26  cm  (see  Figure  D-34c),  but  it  fell 
below  90%  between  0.26  cm  and  0.63  cm.  It  peaked  to  95%  at 
0.65  cm,  then  fell  below  90%  and  reached  a  minimum  of  81%  at 
a  crack  length  of  1.12  cm.  When  the  results  of  five  eddy 
current  operators  were  merged  (subset  No.  35)  and  calculated 
with  the  OSP  scheme,  the  POD(CL95)  reached  92%  at  a  crack 
length  of  0.32  cm  (see  Figure  D-35c).  But  like  the  ultrasonic 
and  penetrant  results,  the  POD(CL95)  for  the  eddy  current 
fell  below  90%  at  higher  crack  length  and  it  reached  a 
minimum  of  81%  at  0.88  cm. 

Data  subsets  36  through  47  (Figures  D-36  to  D-  47) 
from  the  B-1  NDI  demonstration  program.  These  data  show  that 
the  POD(CL95)  either  increases  monotonically  or  remains 
constant  with  increasing  crack  length  once  the  threshold  level 
is  reached.  In  several  data  sets  no  further  measurements  were 
made  once  the  desired  threshold  level  was  demonstrated.  It  is 
interesting  that  the  threshold  level  as  well  as  a  large  portion 
of  the  POD(CL95)  curve  does  not  differ  greatly  for  several  t5q)es 
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of  penetrant  applied  to  materials  such  as  Ti-6A1-4V,  7075-16511 
Al,  PH  13-8  Mo  steel,  4330V  steel,  and  PH  17-4  steel. 

Data  subset  48  represents  the  merged  results  of  four 
data  sources  (RI- Space  Division,  Martin  Marietta,  General 
D5mamics  Convair,  and  RI-B-1)  on  2219-T87  Al  by  the  liquid 
penetrant  method  on  flat  plates. 

Data  subsets  49  through  56  were  obtained  with  fatigue 
cracks  in  4330V  steel  and  7075-T6511  Al  having  a  cylindrical 
shape  (7.62  cm  in  diameter  and  .62  cm  thick).  Data  were  taken 
in  both  the  laboratory  and  production  environments.  None  of 
the  curves  for  these  8  sets  of  data  attained  POD90(CL95), 
however,  the  penetrant  results  for  7075-T6511  Al  did  demonstrate 
POD87(CL95)  at  a  crack  length  of  1.0  cm,  POD87(CL95)  at  a  crack 
length  of  1.17  cm,  and  POD89(CL95)  at  1.47  cm.  The  POD  for 
ultrasonic  shear  wave  reached  a  value  of  P0D86(CL95)  at  a 
crack  length  of  .80  cm  and  POD87(CL95)  at  a  1.27  cm. 

Data  subsets  57,  58,  and  59  were  obtained  with  eddy  current 
inspection  of  bolt  holes  during  tear-down  inspection  of  KC-135 
wings  at  the  depot  level.  The  same  holes  were  Inspected  by  five 
separate  inspection  teams.  The  inspection  results  from  four  of 
'the  five  teams  were  about  the  same.  Figure  D-57  shows 
the  results  of  team  number  two  which  is  representative  of  the 
four  teams.  Figure  D-58  shows  that  the  detection  relia¬ 
bility  of  team  nxamber  four  is  considerably  different  than  the 
other  fcur  teams.  Figure  D-59  shows  the  merced  results 
of  the  five  teams.  None  of  the  five  teams  achieved  PGD90(Gh95)  . 

Data  subsets  60  through  112  (Figures  D-60  to  D-112)  were 
obtained  from  the  Practical  Sensitivity  Limits  program  which  Was 
conducted  by  Boeing  Commercial  Airplane  Company  and  supported 
by  the  NDE  Branch  of  Air  Force  Materials  Laboratory.  The 
inspection  results  were  obtained  in  both  the  laboratory  and 
production  lines  of  several  Boeing  Divisions.  Divisions  used 
different  NDI  procedure  or  specification,  as  well  as  diffeteht 
NDI  techniques.  For  example,  ultrasonic  shear  and  surface  wavea 
of  5  and  10  MHz  were  used,  different  liquid  penetrant  systems  ih 
either  the  Group  V  or  VI  category  were  used,  etc.  Because  the 
results  were  obtained  under  different  NDE  procedures,  it  is 
difficult  to  make  comparison  between  results  from  different 
data  sources.  Furthermore,  the  majority  of  the  defects  used  in 
this  program  were  forging  cracks  instead  of  fatigue  cracks, 
nevertheless  the  length  of  the  crack  at  POD90(CL95)  found  by 
each  of  the  NDI  techniques  is  comparable,  in  most  cases,  to 
those  found  at  other  companies  using  fatigue  cracks. 


6.2  Statistical  Analysis  Procedures 


All  NDE  measurements  are  considered  to  be  binomial  and 
describable  by  binomial  distribution  functions  and  binomial 
statistics.  Because  of  this  consideration,  lower  one  sided 
confidence  limits  and  data  deficiencies  can  be  rigorously 
computed. 

The  only  factor  which  reduces  mathematical  rigor  is  the 
crack  length  parameter.  It  is  necessary  to  group  NDE  data 
into  a  size  interval  that  will  contain  a  population  of  29 
successes  without  a  miss  in  order  to  show  95  percent  con¬ 
fidence  that  the  probability  of  detection  exceeds  90  percent. 
If  a  miss  is  encountered,  it  is  necessary  to  have  a  higher 
population  (see  Table  6-1)  in  order  to  demonstrate  P0D90(CL95) 


Table  6-1 

Interval  Sample  Size  and 
Successes  Required  to  Achieve  POD90(CL95) 


No .  of 

Misses 

Interval  Population 
Required,  N 

No.  of 
Successes,  n 

0 

29 

29 

1 

46 

45 

2 

61 

59 

3 

76 

73 

4 

89 

85 

5 

103 

98 

6 

116 

110 

7 

129 

122 

8 

142 

134 

9 

154 

145 

10 

167 

157 

Once  an  acceptable  size  interval  has  been  chosen  that 
will  produce  a  reasonable  population,  there  remains  a  problem 
of  selecting  the  crack  length  within  the  range  at  which  to 
plot  the  computed  POD(CL95) .  For  conservatism,  the  POD  curves 
in  this  report  are  plotted  through  the  maximum  crack  length  in 
the  interval.  The  crack  interval  is  also  indicated  by  a 
horizontal  line  connecting  maximum  and  minimum  crack  sizes. 
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Three  procedures  for  determining  the  probability  of  de¬ 
tection  for  the  lower  one-sided  confidence  level  are  compared 
in  this  report.  Described  earlier  in  Section  4.3,  they  were 
the  range  interval  (RI)  method,  the  optimized  probability 
method  (OPM)  and  the  overlapping  sixty  point  method  (OSP) . 

The  optimized  probability  method  is  preferred  over  the  RI  or 
OSP  but  requires  more  computational  time.  The  main  claim  to 
success  for  the  OSP  method  is  that  it  forces  groupings  to  be 
large  enough  to  provide  the  capability  for  achieving  POD95 
(CL95) .  The  OPM  should  provide  the  upper  envelope  to  both 
the  RI  and  the  OSP  methods. 

6.3  Data  Deficiencies 

The  data  described  in  this  report  represents  more  than 
30,000  measurements,  yet  there  are  many  gaps  in  the  probability 
of  detection  for  different  defect  types,  material  types,  and 
specimen  geometry  by  the  five  NDE  methods  (ultrasonics,  liquid 
penetrant,  eddy  current.  X-ray,  and  magnetic  particle).  The 
most  data  are  available  for  fatigue  cracks  in  flat  plates  of 
2219-T87  aluminum  which  reasonably  characterize  the  reliability 
of  all  five  NDE  methods.  There  are  also  several  sets  of  data 
for  fatigue  cracks  in  alttminum  cylinders,  forging  cracks  in 
extruded  aluminum  (tandem  T  configuration),  and  fatigue  cracks 
in  bolt  holes  of  actual  aluminum  aircraft  parts.  There  is  a 
notable  data  deficiency  for  steel.  Most  of  the  steel  data 
are  for  fatigue  and  forged  cracks  in  cylinders.  There  are, 
however,  four  sets  of  data  for  fatigue  cracks  in  flat  plates 
of  steel.  Only  five  sets  of  data  are  available  for  titanium 
and  all  of  them  were  taken  on  flat  plates. 

With  the  exception  of  the  B-1  NDE  demonstration  data 
most  of  the  data  exhibit  a  gap  over  some  range  of  the  crack 
lengths.  A  perfect  example  of  this  is  the  data  from  Martin 
Marietta,  Rockwell  International-Space  Division,  and  General 
Dynamics -Convair  Division.  All  these  three  companies  made 
measurements  using  the  same  set  of  specimens  which,  contains 
two  gaps.  As  can  be  seen  from  the  crack  size  distribution 
curve  in  Figure  6-1,  there  are  no  cracks  with  length  between 
0.47  to  0.59  cm  and  between  1.03  to  1.17  cm.  Thus,  no  cracks 
can  be  detected  in  these  two  gaps  and  the  probability  of 
detection,  regardless  of  which  cximulative  scheme  is  being  used, 
is  affected.  A  set  of  specimens  without  gaps  in  the  crack 
size  distribution  curve  is  needed  to  study  the  actual  proba¬ 
bility  of  detection  in  future  programs. 
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6.4  Application  to  Fracture  Mechanics 

The  threshold  crack  length  (the  shortest  crack  length 
that  reached  to  POD90(CL95))  that  was  detected  by  the 
different  NDE  techniques  is  given  in  Table  5-1.  However, 
as  can  be  seen  from  the  curves  in  Appendix  D,  the  POD  (CL95) 
in  many  instances  falls  below  90%  for  some  crack  length  longer 
than  the  threshold  level.  Thus  the  potential  user  is  cautioned 
against  using  these  threshold  crack  lengths  in  the  design 
process.  In  the  first  place,  the  POD  can  vary  from  company 
to  company  even  when  the  same  inspection  procedures  were  used 
because  of  training  procedures  and  using  different  inspectors. 
In  the  second  place,  some  of  these  POD  curves  do  not  have  a 
zero  or  positive  slope  for  crack  length  above  the  threshold 
level.  The  decrease  in  the  POD  above  the  threshold  level  could 
be  a  real  phenomenon  and  might  not  necessarily  be  attributed 
to  human  factors.  There  are  scatter  reports,  in  the  case  of 
liquid  penetrant,  that  indicate  penetrant  washes  out  of  wide 
cracks  before  the  developer  is  applied. 

In  the  fracture  mechanics  design  process,  one  must  be 
assured  with  some  probability  at  some  confidence  level  that 
no  cracks  larger  than  the  assumed  size  can  be  present  in  the 
structure.  Thus,  for  fracture  mechanics  to  be  a  viable  design 
process,  the  probability  of  detection  as  a  function  of  crack 
length  must  be  monotonically  increasing  or  at  least  levels 
off  above  some  selected  threshold  crack  length. 
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Niamber  of  Cracks 


6.5  Optimum  Demonstration  Program 

In  order  to  determine  the  reliability  of  an  NDE  technique 
without  operator  influence,  the  optimum  demonstration  program 
will  have  to  employ  computer  automation  in  scanning  specimens 
and  interpreting  NDE  signals.  It  should  incorporate  ultrasonic, 
eddy  current  and  penetrant  inspection  into  each  demonstration 
program. 


The  data  in  this  report  can  be  used  to  estimate  the 
sample  size  required  for  the  optimum  demonstration  program. 

For  example,  if  one  can  afford  to  test  100  specimens  and 
requires  a  50-50  chance  of  success  in  demonstrating  POD90/CL95 
then  one  would  have  to  select  a  data  set  representative  of  this 
expected  capability  and  select  a  crack  length  for  his  demon¬ 
stration^  prog  ram  for  which  the  probability  of  detection  is  at 
least  957o.  If  he  can  afford  only  45  specimens  then  he  must 
choose  a  crack  length  for  which  the  probability  of  detection 
is  at  least  987.  This  can  be  seen  in  Figure  6-5  where  number  of 
misses  corresponding  to  POD90(CL95)  is  plotted  against  the 
number  of  specimens  tested.  The  shaded  region  is  the  region 
of  success  for  which  POD  90(CL95)  has  been  met  or  exceeded. 

The  broken  lines  show  the  number  of  failures  expected  for 
various  values  of  POD  (98,  95,  94,  93,  92,  and  907). 


Assume  that  data  set  #  1  (Figure  D-1)  is 
for  a  geometry  and  NDE  method  that  is  most  representative 
of  those  required  in  a  demonstration  program.  These  data 

demonstrate  a  POp(CL95)  at  .345  cm.  Figure  D-la  would  lead 
one  to  believe  that  POD  90(CL95)  might  be  demonstrated  at 
about  .27  cm  for  which  38  out  of  39  cracks  were  detected. 
Figure  D-la  shows  that  one  can  be  50  percent  confident  that 
the  probability  of  detection  exceeds  957  for  this  case  if 
his  NDE  techniques  are  as  good  as  those  used  to  produce  data 
set  iF  1.  Figure  6-2  shows  that  if  110  test  cracks  were 
fabricated  and  tested  and  that  less  than  5  cracks  were  missed, 
then  the  demonstration  program  would  be  Successful. 


The  previous  example  is  indicative  of  the  usefulness  of 
these  data  as  a  means  for  minimizing  the  number  of  specimens 
required  for  achieving  a  successful  demonstration  program. 

The  common  practice  of  manufacturing  a  large  number  of  crack 
specimens  for  which  there  is  little  chance  to  achieve 
POD  90(CL95)  will  be  eliminated  by  this  approach  to  designing 
an  optimum  demonstration  program. 
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The  Shaded  Region 
Shows  the  Conditions 
for  a  Successful 
Dernonatration  Program 


NUMBER  OF  SPECIMENS  TESTED 


Figure  6-2  Conditions  for  Success  in  Selecting 
a  Sample  Size  that  will  Potentially 
Demonstrate  POD90(CL95) 


VII.  CONCLUSIONS 


This  program  was  sponsored  by  NASA  as  a  service  to  the 
new  fracture  mechanics  approach  to  structural  design  whereby 
system  reliability  may  be  related  to  the  expectation  that 
flaws  larger  than  a  specific  size  may  exist  in  the  structure. 

The  program  objectives  were  to  (1)  collect  all  available  NDE 
reliability  data,  (2)  examine  the  validity  of  the  data, 

(3)  select  and  computerize  a  statistical  procedure  for 
analyzing  the  data,  and  (4)  generally  assess  the  current  state- 
of-the-art  in  NDE.  We  feel  that  the  surface  has  been  broken 
by  this  program  but  vigorous  follow-on  work  would  be  very 
valuable  to  the  field  of  structural  design. 

Some  of  the  specific  conclusions  that  can  be  drawn  from 
the  contents  of  this  report  and  the  experience  gained  in 
performing  this  study  are : 

(1)  The  human  factors  influence  stands  out  as  dominating  in 
influencing  NDE  reliability.  Although  a  company  can  demonstrate 
a  90  percent  probability  of  detecting  a  crack  at  a  lower  one 
sided  confidence  limit  at  the  95  percent  confidence  level,  there 
is  no  guarantee  that  this  is  indicative  of  a  continuing  capa¬ 
bility  unless  some  form  of  automation  has  replaced  human  judgment. 
This  then  indicates  a  need  for  some  type  of  periodic  audit  of 
capability. 

(2)  Although  more  than  30,000  measurements  have  been  filed  in 

a  computer  library  and  analyzed  to  determine  the  NDE  reliability 
of  X-ray,  ultrasonics,  eddy  current,  and  penetrants,  there  are 
still  many  gaps  in  the  data:  There  are  no  eddy  current  data 
in  the  data  bank  for  titanium  nor  are  there  much  data  for  crack 
lengths  near  .5  or  near  1.2  cm.  These  data  deficiencies  cause 
large  uncertainties  in  the  POD  plots  for  these  crack  lengths. 

(3)  Binomial  statistics  are  recommended  for  analyzing  NDE 
reliability  data.  This  report  contains  a  comprehensive  dis¬ 
cussion  of  the  preferred  analysis  model. 

(4)  The  optimized  probability  method  (0PM)  of  data  cumulation 
is  preferred  over  the  range  interval  or  the  overlapping  sixty 
point  scheme  for  computing  probability  of  detection.  The 
optimum  probability  model  which  was  developed  during  this 
program  is  discussed  in  this  report. 
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(5)  Data  show  that  the  X-ray  technique  is  not  viable  for 
detecting  tight  fatigue  cracks. 

(6)  In  order  to  obtain  optimum  results  from  an  NDE  demon¬ 
stration  program,  special  care  must  be  taken  in  the  design 
of  flaw  size  distribution.  The  cumulative  distribution  must 
always  maintain  a  positive  slope  or  there  will  be  gaps  without 
data  which  can  have  deleterious  effects  on  the  POD  curves  as  a 
function  of  flaw  sizes. 

(7)  The  data  presented  in  this  report  are  offered  as  a  guide 
for  manufacturers  who  intend  to  perform  an  NDE  reliability 
demonstration  program  and  can  identify  data  sets  that  represent 
their  expected  NDE  capabilities.  However.,  due  to  the  human 
factor  influence  on  POD,  the  data  cannot  be  used  as  design 
information.  Each  company  must  develop  reliability  numbers 
for  their  own  use. 

(8)  A  continuing  program  to  maintain  an  NDE  reliability  data 
file  and  to  optimize  the  data  processing  procedures  formulated 
during  this  program  would  be  useful  to  the  field  of  fracture 
mechanics.  As  the  data  base  becomes  larger  it  will  begin  to 
support  parametric  studies  of  the  influence  of  NDE  variables. 
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APPENDIX  A 


A  MODEL  TO  GRADE  THE  QUALITY  OF  THE  DATA  SETS 


The  thoroughness  of  the  characterization  of  the  defect- 
specimen  and  the  documentation  of  the  inspection  procedures 
affects  the  quality  and  the  usefulness  of  the  data  sets,  A 
designer  will  not  have  confidence  in  using  the  data  unless 
the  inspection  procedures  and  defect  dimensions  are  sufficiently 
documented  so  that  it  can  be  reproduced  in  future  inspections. 

The  model  described  in  this  section  will  only  relate  to  the 
quality  of  each  set  of  data.  It  will  not  address  the  question 
of  applicability.  That  is,  a  set  of  data  obtained  on  flat 

plates  will  not  be  graded  on  the  basis  of  its  applicability 
to  the  design  of  complex  structures. 

The  model  is  empirical  and  the  weighting  factors  assigned 
to  the  various  known  pertinent  parameters  are  rather  arbi¬ 
trary.  However,  it  does  represent  a  first  attenq>t  to 
quantitatively  evaluate  the  quality  of  a  data  set.  Each  of 
the  pertinent  parameters  in  this  model  is  listed  in  the 
Input  Data  Format  which  will  be  discussed  in  the  next  sub¬ 
section.  The  grade  for  a  given  set  of  data  can  be  tallied 
in  the  computer  by  checking  the  entries  to  the  columns  con¬ 
taining  these  pertinent  parameters.  A  score  of  one  hundred 
(100)  corresponds  to  a  perfect  set  of  data.  A  perfect  set 
of  data  is  one  where  all  the  pertinent  parameters  are 
documented. 

The  preliminary  model  to  grade  the  data  quality  for  the 
ultrasonic,  eddy  current,  liquid  penetrant,  magnetic  particle, 
and  X-ray  methods  is  given  in  Table  Ar-1.  The  model  is  divided 
into  two  major  groups.  Group  A  is  the  characterization  of 
specimens  and  defects  and  it  is  common  to  all  techniques. 

Group  B  is  the  documentation  of  the  inspection  and  is  different 
for  different  NDE  techniques.  The  first  eight  parameters  in 
this  last  group  are  common  to  all  techniques.  Each  technique 
has  about  eleven  parameters  that  are  considered  pertinent 
to  adequately  document  the  inspection.  Parameter  A1  is  the 
description  of  the  specimen  geometry  and  defect  location. 

This  is  considered  a  key  parameter  and  a  value  of  7  points 
is  arbitrarily  assigned  to  it.  If  no  description  about  the 
specimen  geometry  and  defect  location  is  given,  that  data 
point  or  set  of  data  will  receive  no  points.  Those  parameters 
that  are  marked  by  an  asterisk  are  considered  key  parameters. 

If  any  of  them  is  not  recorded,  the  data  point  or  set  will  be 
considered  for  possible  exclusion  from  statistical  analysis. 
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Table  A-1 


THE  GRADING  FACTORS  USED  TO  GRADE  NDE  DATA  QUALITY 


A.  Specimen  -  Defect  Characterization  information 

1.  Specimen  geometry  complexity  and  defect  * 

location 

2.  Ctack  dimension  verification  * 

3,  Defect  t37pe  * 

4.  Specimen  surface  condition  * 

5,  Material  characterization  (thermo¬ 
mechanical  history) 

6,  Material  inside  defect  * 
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7 

10 

3 

3 

3 

3 


B.  Inspection  Docvimentation  65 

1.  False  indication  recording  4 

2.  Inspector  qualification  level  3 

3.  Knowledge  of  defect  orientation,  *  9 

location,  and  presence  by  inspector 

4.  Use  of  proof  load  *  3 

5.  Use  of  control  specimens  without  3 

crack 


6.  Method  of  data  recording  3 

7.  Method  of  scanning  specimens  3 

8.  Insp.  environment  *  3 

9.  Number  of  insp.  prior  to  this  insp,  3 

10..  Parameters  recorded  by  nondestructive 

testing  standards 

a.  Radiography  38 

1)  Radiographic  source  *  3 

2)  Ref.  standard  3 

.4-2 


1.0  . 


(Continued) 


3)  Detector  t3rpe  * 

4)  Voltage  * 

5)  Current  * 

6)  Exposure  time  * 

7)  Source/film  distance  * 

8)  Angle  of  entry  * 


9)  Film  development  parameters 

10)  Radiograph  density  * 

11)  Radiographic  equipment  type 
b«  Ultrasonics 

1)  Ultrasonic  method  * 

2)  Frequency  * 

3)  Transducer  type  and  size  * 

4)  Reference  standard  type  and  size  * 

5)  Angle  of  incidence  (inside  the  * 
material) 

6)  Equipment  t37pe 

7)  Gate  alarm  level  (%  of  ref,  signal)  * 

8)  Gain  setting  (%  of  screen  saturation) * 

9)  Type  of  coupling 

10)  Index  interval  * 

11)  Contact  or  Immersion 


4 

4 

4 

4 

4 

3 

3 

3 

3 
38 

4 
4 
4 
4 
3 

3 

4 
4 
3 
3 
3 
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10.  (Continued) 

c.  Eddy  Current 

38 

1) 

Method  of  Scan 

3 

2) 

Coil  size 

* 

4 

3) 

Coil  arrangement  and  shape 

3 

4) 

Frequency 

* 

4 

5) 

Reference  t3q)e  and  size 

4 

6) 

Equipment  t3q)e 

3 

7) 

Index  interval 

* 

3 

8) 

%  of  meter  response 
(Real  Part) 

4 

9) 

%  of  meter  response 
(Imaginary  Part) 

* 

4 

10) 

Lift-off  compensation 

3 

11) 

Signal  processing 

3 

d.  Penetrant 

38 

1) 

Penetrant  tj^pe 

* 

4 

2) 

Developer  type 

4 

3) 

Classification  of  penetrant 
no.) 

(group 

2 

4) 

Emulsifier  t5q)e 

* 

4 

5) 

Pre-insp,  surface  cleaning  and  * 
penetrant  removal 

3 

6) 

Method  of  penetrant  application 

3 

7) 

Dwell  time 

* 

4 

8) 

Development  time 

A-4 

* 

4 

10,  (Continued) 

9)  Wash  time 

10)  Light  type  and  intensity  at 
specimen  surface 

11)  Reference  standard  t3^e 
e.  Magnetic  Particle 

1)  Type  of  current  used 

2)  Current  level  (Amperes)  . 

3)  Method  of  magnetization 

4)  Direction  of  magnetization 

5)  Magnetic  flux  density 

6)  Magnetic  particle  t57pe  and  size 

7)  Magnetic  particle  density 

8)  T3pe  of  liquid  vehicle 

9)  Method  of  particle  application 

10)  Equipment  tjpe 

11)  Dwell  time  (seconds) 
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DATA  POOLING 


B,1  Data  Pooling  by  NDE  Method  and  Parameters 

Data  that  meet  the  preliminary  criterion  as  described  in 
Subsection  3.1  are  input  to  the  computer  for  statistical 
analysis.  Then  data  from  several  sets  are  pooled  and  analyzed 
if  they  have  a  common  set  of  parameters.  Data  from  different 
NDE  methods  are  not  to  be  pooled.  For  a  given  NDE  method  such 
as  ultrasonic  shear  wave  at  5  MHz,  reliability  curves  will  be 
plotted  for  a  material  tj^e,  a  defect  t37pe,  an  environment,  a 
specimen  geometry  and  defect  location,  and  either  before  or  after 
some  enhancement  such  as  a  proof  test.  Composite  reliability 
curves  can  be  plotted  by  pooling  data  with  different  parameters, 
such  as  pooling  5  and  10  MHz  shear  wave  data,  laboratory  and 
production  data,  flat  plate  and  cylindrical  shell  data,  etc. 

B.2  Statistical  Pooling  Criteria 

The  NDE  data  that  is  compiled  for  this  contract  was 
collected  from  different  sources  using  different  calibration 
factors,  different  equipment,  different  personnel,  different 
environments,  etc.  Each  set  of  data  therefore  contains  unique 
source  characteristics  that  preclude  indiscriminate  pooling  for 
reliability  calculations. 

A  statistical  pooling  criteria  has  been  developed  to 
safeguard  against  mistakes  or  inconsistencies  in  the  data 
which  would  produce  abnormal  statistical  results.  The  data 
pooling  technique  is  based  solely  on  the  binomial  distribution 
and  is  described  below  as  a  procedure  which  can  be  imp lamented 
on  a  computer.  The  procedure  consists  of  the  following  four 
steps . 

(1)  The  best  single  estimate,  p  ,  for  probability  of 
detection  ^ 


(9) 
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where  M  is  the  number  of  data  sets  to  be  pooled, 

(2)  The  best  single  estimate  of  the  true  probability 


-  \ 
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(10) 


of  each  of  the  data  subsets  is  computed. 

(3)  Consider  the  binomial  distribution  function 
for  each  data  set  n0  having  a  true 

probability  of  detection  given  by  The 

two-sided  probability,  a2»  that  (N|^,  nu) 
and  all  less  likely  outcomes  are  possible  is 
computed  from 


(4)  All  data  sets  havil^ng  a  value  of  an  less  than  a 
reference  value  a  (computer  input  value)  are 
removed  as  candidates  for  pooling. 


The  choice  of  cc  is  somewhat  arbitrary  and  depends  upon 
the  acceptable  risk.  The  data  sets  that  will  be  rejected 
from  pooling  for  a  given  a‘  value  will  be  reviewed.  If  no 
abnormalties  are  found  within  each  set  of  data  (l,e,,  no 
mistakes  in  data  recording,  or  other  possible  means  of 
causing  the  probability  of  detection  to  be  normally  high  or  low) 
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a  new  value  of  oe'  will  be  tried  and  one  that  will  permit 
the  data  sets  to  be  pooled  with  the  data  base  will  be 
selected.  Thus,  the  value  for  ct'  may  be  governed  by  operator 
judgment  of  the  validity  of  the  data. 

Table  B-1  is  an  example  which  contains  six  hypothetical 
binomial  experiments  (assuming  all  sets  have  the  same  measure¬ 
ment  parameters),  (For  this  example  a'  was  chosen  to  be  ,05), 
The  ot2  values  for  sets  C,  E,  and  F  are  very  low.  The  ot2 
values  for  sets  C  and  E  were  calculated  using  Equation  (11) 
and  for  set  F  was  calculated  using  Equation  (12) ,  For  set  C 
one  is  only  2,22%  confident  that  one  out  of  eight  measurements 
is  successful.  For  set  E  one  is  only  0,86%  confident  that  one 
out  of  ten  measurements  is  successful.  For  set  F  one  is  only 
0,11%  confident  that  seven  out  of  seven  measurements  are 
successful.  The  confidence  is  too  low  for  measurements  to  be 
pooled  with  those  of  sets  A,  B,  and  D, 

Upon  rejecting  sets  C,  E,  and  F,  new  a2  values  are 
calculated  for  sets  A,  B,  and  D  using  the  p^  (16/40),  These 
three  sets  have  comparable  confidence  limits  and  they  will  be 
pooled. 
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Table  B-1 

SIX  SETS  OF  DATA  FROM  DIFFERENT  SOURCES 
(A-F)  TESTED  FOR  POOLING 


Source 

N  Number  of 
Measurements 

n  Number  of 
Successes 

Ot,  (Alpha) 

A 

24 

9 

.4085  (accepted) 

B 

8 

3 

.3612  (accepted) 

C 

8 

1 

.0222  (rejected) 

D 

8 

4 

.3572  (accepted) 

E 

10 

1 

.0086  (rejected) 

F 

7 

1 

7 

.0011  (rejected) 

TOTAL 

65 

25. 

A 

24 

9 

i 

.3641  (aGcepted) 

B 

8 

3 

.3361  (accepted) 

D 

8 

4 

.3834  (accepted) 

TOTAL 

40 

16 

B-4 
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NASA.  FTN  COMPUTER  CODE 

NASA.  FTN  is  written  in  Fortran  and  organized  with  a 
main  routine  performing  all  executive  functions.  Main  does 
no' data  manipulation,  it  simply  calls  other  routines  which 
manipulate  the  data. 

The  flow  diagram  Figure  C-1  illustrates  the  run  sequence 
of  the  program.  Two  inputs  required  for  the  program  are  the 
Master  Data  File  and  a  Problem  Input  Deck.  At  the  end  of  a 
production  run,  the  program  lists  the  computational  results. 

C-1  Master  Data  File 


The  master  data  file  consists  of  a  sequence  of  'data  sets!. 
Each  data  set  grouping  has  a  common  combination  of  conditions 
under  which  a  series  of  specimens  were  inspected.  These  con¬ 
ditions  are  described  in  two  256  word  records  (header  records) . 
The  results  of  the  inspection  are  called  data  points  and  are 
stored  after  the  two  header  records.  Each  data  point  is  8  words 
long  and  contains:  crack  ID,  detect  code,  crack  length,  crack 
depth,  measured  crack  length,  measured  crack  depth,  surface 
finish  and  thickness.  The  'data  records'  consists  of  256  words 
which  contain  32  data  points  in  each.  Each  data  set  is  labeled 
by  a  number  which  is  unique  to  the  data  set.  The  data  set  number 
and  the  number  of  data  records  per  data  set  are  stored  in  the 
header  records  of  each  data  set.  The  master  data  file  is  stored 
on  magnetic  tape  in  binary  format,  256  words  per  record. 

C-2  Program  Flow  (Ref.  to  Table  C-1  ) 

The  first  step  in  running  the  program  is  to  read  all  the 
program  options.  The  subroutine  then  reads  the  maister  data 
file  to  extract,  data  points  for  analysis.  These  data  are 
stored  on  a  local  disk  file  for  up  to  10,000  data  points.  Each 
data  point  on  this  data  file  is  described  by  four  words,  crack 
size,  detect/no  detect,  crack  ID,  and  data  set  number.  The 
input  options  have  already  (either  implicitly  or  explicitly) 
described  a  set  of  common  conditions.  Subroutine  Order  then 
rewrites  the  local  data  file  in  increasing  crack  size. 
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Figure  G^l  Structure  of  Main  Routine  in 
NASA.  FTN  Code 


Table  C-1 


Glossary  of  Programs  and  Subroutines 
Used  in  NASA.  FTN 


NASA 

ENTER 

SORT 

ORDER 

POOL 

POOLR 

RJCT 

PCK 

HSTGRM  - 
OVLP30  - 

SMOTHG  - 


Main  program  to  control  the  program  steps  flow 
in  Figure 

Routine  to  read  in  cards  which  contain  the 
program  options.  The  options  are  stored  in  commons 
for  use  by  other  routines. 

Routine  that  reads  the  master  data  file,  sorts  out 
data  for  analysis,  then  writes  this  data  on  local 
disk  file. 

Routine  to  sort  the  data  on  the  local  data  file  so 
that  the  order  of  the  data  will  be  by  increasing 
crack  size. 

Routing  to  determine  whether  a  particular  data  set 
belongs  to  the  rest  of  the  sets  being  considered. 

Routine  to  calculate  probability  of  data  sample 
belonging  to  a  group  having  a  known  probability 
of  detection. 

Routine  to  zero  out  data  points  of  a  data  set  that 
has  been  rejected  on  statistical  grounds. 

Routine  to  remove  'holes’  in  the  local  data  file 
that  are  created  by  RJCT  and  to  form  a  packed  data 
file. 

Routine  to  sort  data  into  32  size  interval  groups. 

Routine  to  perform  an  analysis  of  the  data  by  using 
the  procedure  known  as  the  "OSP"  scheme. 

Routine  to  perform  an  analysis  of  the  data  by 
using  the  procedure  known  as  the  0PM. 
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STRGHT 

BIN 

DEFJC 

LST 

CKNRT 


Table  C-1  (Continued) 


Routine  to  perforin  analysis  of  the  data  fcy  using 
the  procedure  known  as  the  "RI"  scheme. 

Routine  to  calculate  POD  at  a  specific  confidence 
limit  for  a  binomial  experiment  consisting  of  N 
measurements  and  n  successes.. 

Routine  to  calculate  the  number  of  successful 
experiments  which  must  be  performed  in  order  to 
know  the  lower  limit  probability  of  detection 
within  a  given  confidence  limit. 

Routine  to  list  the  results  of  the  analysis. 

-  Routine  which  performs  paging  of  the  local  data 
file. 
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If  pooling  is  desired,  routine  POOL  can  be  used  to 
calculate  the  probability  that  each  data  set  statistically 
belongs  to  the  rest  of  the  data  sets  that  are  collected 
together.  Any  data  set  that  gets  rejected  is  removed  from 
the  local  data  file  by  zeroing  the  data  points.  Routine  PCK 
takes  the  local  data  file  and  re-packs  it  so  that  there  are 
no  zeroed  points.  All  data  sets  removed  are  listed  on  the 
printer.  The  method  of  analysis  determines  the  route  the 
program  takes  next.  If  'overlapping  60'  is  chosen  routing 
OVLP30  "OSP"  takes  the  local  data  file  and  processes  the 
data  directly.  Routing  HSTGRM  is  used  to  divide  the  data 
points  into  32  groups  which  have  equal  crack  size  intervals 
ranging  between  a  minimum  and  maximum  input  crack  size.  All 
crack  sizes  falling  below  the  minimum  are  included  in  the 
first  interval  and  all  data  falling  above  the  maximum  are 
included  in  the  thirty  second  interval.  Either  the  routine 
STRGHT  "RI"  or  the  SMOTHG  "OPM"  analysis  procedure  can  be 
selected  to  analyze  the  data.  The  results  are  then  listed. 

The  program  then  returns  to  the  beginning  to  read  more  cards. 

If  there  are  no  more  problems  to  be  solved,  the  program  terminates. 

C-3  Sorting  Option 

The  program  has  two  sort  options.  The  first  option  uses  the 
data  set  nimiber  directly.  (See  Figure  4-2  )  .  The  user  specifies 
the  number  of  data  sets  to  be  considered,  then  specifies  the  data 
set  numbers.  The  second  option  (Figure  4-3) j  makes  use  of  the 
header  card  information  which  includes  26  words  or  numerical 
codes  for  the  alphanumeric  data  that  is  contained  in  the  header 
records.  By  supplying  values  for  these  words,  each  header  record 
can  be  compared  to  this  'master'  input.  No  zero  values  are 
compared.  Each  time  a  match  is  made,. ..that  data  set  is  stored 
on  the  local  data  file.  A  maximum  of  5  master  header  records  can 
be  combined  per  run. 

C-4  Program  Limitations 

The  maximum  crack  size  that  can  be  stored  and  analyzed  is 
2.54  cm.  The  maximum  number  of  data  points  that  can  be  combined 
into  a  set  is  1000.  The  maximum  number  of  measurements  that  can 
be  analyzed  by  the  subroutine  "DEFIC"  which  computes  data 
deficiencies  is  120. 
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Figure  c-2  Sort  Subroutine  Flov>  When  Using  Data  Set 
Number  Only  for  Sorting 
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C-5  Local  Data  File 


The  local  data  file  is  a  direct  access  file  which 
occupies  a  continuous  block  or  disk  space.  A  paging  method 
has  been  incorporated  to  avoid  executive  disk  l/O.  The 
subroutine  CKNXT  is  used  for  paging.  At  any  given  time,  1024 
data  points  may  be  held  in  the  core,  therefore  a  disk  access 
does  not  have  to  be  made  each  time  a  new  data  point  is  addressed. 
The  local  data  file  has  ten  pages  available  for  program  use, 

C-6  Ordering  Method 

A  subroutine  names  "ORDER"  reorders  the  local  data  file 
entries  in  increasing  crack  length  then  rewrites  the  ordered 
data  file  in  place  of  the  original  unordered  local  data  file. 

This  ordered  data  is  then  used  as  input  for  data  pooling  and 
statistical  analysis, 

C-7  Lower  One-Sided  Confidence  Limits  Calculations 

The  binomial  Equation  (5)  is  solved  by  setting  G  equal 
to  the  confidence  level  desired  and  determining  the  value  of 
Pj^  which  satisfies  the  equation.  A  Fortran  IV  program  which 
has  been  made  operable  in  a  PDO  11/45  computer  to  perform  this 
function  is  given  in  Figure  4-5.  p^  is  the  probability  of 
detection  at  the  lower  confidence  limit. 

The  procedure  used  to  determine  how  many  more  measure¬ 
ments  are  required  in  each  interval  for  the  95%  confidence 
curve  to  reach  the  90%  probability  of  detection  level  is  to 
solve  Equation  8  for  and  .  Figure  4-6  gives  the  listing 
of  a  Fortran  IV  program  to  perform  this  calculation. 
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Figure  C~3  Sort  Subroutine  Flov/  When  Using  Master  Control 
Records  to  Sort  on  Master  Data  File 
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SUBROUT  INF.  B I N<  AR  i ..  BR2 ..  AR3 ..  AR 1 0  ) 

1  lF<f^R2)2.2.>4 
AR 10=0.0 
RETURN 

IF<0R2~AR1>7.5,.5 
OR  1 0=<  1 . 0-OR3  ytU  1 . 0/OR 1 ) 

RETURN 
OTT=2.03»:AR2 
IF<0TT~0R1>9..9.12 
OR4=AR2-i.0 
OR5=-1.0 
GO  TO  15 
0R4=AR1-AR2 
0R3=1 . 0-AR3 
AR5=“1.0 
ORi0=e.5 
OR6=1.0 
ORS=0 . 0 
AR9=1 .0 
ARil=ARl 

AR7=<  ^RWU^RB  ytXCl.  0-AR10  >tt(  AR1~0R8  )  > 
IF'C0R8-AR4  >22/ 27..  22 
0R8=AR8+1 .0 
AR9=AR9^>:AR11/0R8 
ARll— AR11“1  0 

AR7=AR7+AR9:i:<AR10:»::|:AR8>*<<  1 .0-AR10>»4:(  AR1-AR8>> 
GO  TO  21 

IFC  AR3~AR7  >28/ 28/ 30 
AR20=AR10-AR5-'<  2 . 0»1:<  AR64-1 . 0  )  > 

GO  TO  31 

AR20=AR10+AR5/<  2 . 0tt< AR6+1 . 0  >  > 

CC:C=ABS<AR7-AR3) 

IF<  CCC-e . 0001 >36/ 36/ 33 
AR6=AR6+1 .0 
AR10=AR20 

GO  TO  17 

1F(ATT“0R1>6.6.37 

OR10=1 .0-AR10  \ 

RETURN  I  \ 

39  END 


Figure  0-4  Computer  Code  Used  to  Calculate  Lower  One-Sided 
Confidence  Limit 
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Figure  C-5  Computer  Code  Used  to  Calculate  Additional  Measurements 
Required  to  Achieve  90%  Probability  of  Detection  at 
95%  Confidence  Limit 


e-8  DATA  INPUT  FORMAT 

The  data  are  filed  in  mass  memory  by  using  a  'data  set’ 
concept.  All  the  common  control  parameters  for  a  group  of 
data  points  are  loaded  together.  There  are  26  such  control 
parameters.  The  first  14  are  the  same  for  all  NDE  methods, 
the  next  12  are  different  for  each  of  the  five  NDE  methods. 
The  results  of  the  inspection  follow  the  control  parameters. 
Eight  ntimbers  are  used  to  describe  each  point.  (Crack  ID, 
detect  code,  crack  length,  crack  depth,  measured  crack 
length,  measured  crack  depth,  surface  finish,  thickness). 

A  typical  control  parameter  listing  is  shown  in  Table 
C-1.  Nine  hundred  eighty  four  flaw  measurements  have  these 
parameters  in  common.  The  data  in  Table  C-1  can  be  compacted 
for  computer  storage  into  the  format  shown  in  Table  C-2.  A 
special  data  set  listing  like  that  shown  in  Table  C-1  can  be 
prepared  from  the  line  of  integers  shown  in  Table  C-2  by 
using  the  parameter  key  included  at  the  end  of  this  appendix. 
Four  of  the  984  data  points  for  data  set  #  1  were  entered 
onto  each  of  246  lines  in  the  format  depicted  in  Table  C-3, 
Each  complete  data  set  is  terminated  by  a  0000  entry  in 
Columns  1  through  4  of  the  digital  computer  data  sheet. 
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Table  C-1  Data  Set  Parameters 
for  Data  Set  Nximber  1 
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TaBle  C-2  Data'  Set:  fhtry  for  ebmpact  Storage  of 
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Table  C-3  Flaw  Size  Data  Entry  Format 
Showing  Last  Four  Entries  with  End  of 
File  Designated  by  "0000 ” 
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DEFECT  TYPE  OPER#^TGR  ID 
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QUALIFICATION  INSP  ENUIRONMENT 
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RECORD  TYPE 
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DEFECT  MATERIAL 
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APPENDIX  D 


COMPUTER  OUTPUT  -  NDE  RELIABILITY  DATA 

The  data  for  Figures  D-la,  D-lb,  and  D-lc  were  obtained 
with  an  ultrasonic  surface  wave  technique  and  at  a  frequency 
of  10  MHz.  The  abscissa  is  the  surface  length  of  a  fatigue 
crack.  The  NDE  technique  (ultrasonics)  is  given  on  the  upper 
left  hand  corner  of  the  figures  immediately  after  the  date 
when  the  curves  were  plotted.  Test  1,  Test  2,  or  Test  3  is 
written  on  the  same  top  line  of  the  page.  Test  1  refers  to 
the  range  scheme  where  the  lower  one-sided  confidence  limit, 

P]^,  was  calculated  for  equal  flaw  size  intervals.  In 

Figure  D-la  the  flaw  size  interval  is  0.038  cm. ^  Test  2  (Figure 
D-lb)  refers  to  the  use  of  the  optimized  probability  method  to 
calculate  the  p^^.  Test  3  (Figure  D-lc)  refers  to  the  use  of 
the  overlapping  sixty  point  scheme  to  calculate  the  p]^.  The 
top  line  of  the  computer  output  also  contains  the  name  of  the 
company  or  the  agency  that  published  the  report  from  which  the 
data  were  taken.  In  Figure  D-1  the  company  is  Martin  Marietta 
of  Denver,  Colorado,  The  test  specimen  martial  type  is 
identified  in  the  figure  caption  on  the  bottom  of  the  page  or 
in  Table  5-1  of  Section  V  where  the  data  set  number  can  be 
matched  with  the  number  in  parenthesis  in  the  upper  right 
hand  side  of  each  figure. 

The  first  column  of  the  tabulated  results  in  these 
figures  lists  the  range  numbers.  The  second  column  gives 
the  minimum  crack  length  of  each  range  and  the  third  column 
gives  the  maximum  crack  length  of  each  range.*  The  fourth 
column  gives  the  total  number  of  measurements  or  observations 
(N)  for  each  size  range  and  the  fifth  column  gives  the  number 
of  detections  (n)  for  each  size  range.  The  probability  of 
detection  at  50%  and  95%  confidence  level**  is  given  in  the 
sixth  and  seventh  columns  respectively.  The  last  two  columns 
list  the  number  of  new  measurements  that  must  be  made  with  no 
misses  and  with  one  miss,  respectively,  to  achieve  90% 
probability  of  detection  at  a  95%  confidence  level.  The  zeroes 
in  these  last  two  columns  indicate  that  either  no  measurements 
need  to  be  made  or  the  number  is  too  large  for  practical 
consideration. 

*  Tabulated  data  give  crack  lengths  in  mils  while  the  graphs 
give  both  mils  and  cm  scales.  This  dual  labeling  is  offered 
as  a  convenience  for  the  design  engineer. 

**  Hereafter  POD90(CL95)  will  be  used  to  refer  to  a  95  percent 
confidence  that  the  true  probability  of  detection  equals  or 
exceeds  90  percent. 
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All  the  reliability  curves  discussed  in  this  appendix 
are  presented  in  the  above  format.  Additional  information 
such  as  flaw,  type,  material  alloy  tjrpe,  contract  or 
report  identification  number,  and  other  pertinent  parameters 
associated  with  the  data  used  to  generate  these  reliability 
curves  and  tables  are  included  wherever  possible. 
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Figure  D-1  Probability  of  Detection  for  2219-T87  Al  Using 
Ultrasonic  Surface  Waves,  Fatigue  Cracks  in 
Flat  Plates.  Lab,  Env. 
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Figure  D- 2  Probability  of  Detection  for  2219-T87  Al  Using 

Liquid  Penetrant,  Fatigue  Cracks  in  Flat  Plates. 
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(c)  Overlapping  Sixty  Point  Method  of  Data  Cumulation 
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Figure  D-12  (Concluded) 
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(a)  Range  Interval  Method  of  Data  Cumulation 
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Figure  D-13  (Continued))  ' 
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(c)  Overlapping  Sixty  Point  Method  of  Data  Ciamulation 
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(a)  Range  Interval  Method  of  Data  Cumulation 
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(b)  Optimum  Probability  Method  of  Data  Cumulation 
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(c)  Overlapping  Sixty  Point  Method  of  Data  Cumulation 
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(a)  Range  Interval  Method  of  Data  Cumulation 
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(b)  Optimum  Probability  Method  of  Data  Cumulation 
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(c)  Overlapping  Sixty  Point  Method  of  Data  Cumulation 
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(a)  Range  Interval  Method  of  Data  Cumulation 


0^- JUL - .  5  RrtD I QGRrtPHY 

RANGE  MIN  LN  NrtK  LH 

1  7  *  02* 

2  25  .V  ,''36: 


38 

52 

4 

54 

67 

5 

68 

82 

6 

83 

97 

7 

98 

111 

8 

115 

126 

9 

129 

141 

10 

143 

157 

11 

158 

171 

12 

182 

185 

13 

190 

197 

14 

0 

0 

15 

0 

0 

16 

241 

247 

17 

248 

262 

18 

268 

275 

19 

279 

290 

20 

295 

306 

21 

310 

322 

22 

323 

336 

23 

338 

352 

24 

356 

362 

25 

370 

381 

26 

384 

393 

27 

408 

408 

28 

426 

426 

29 

442 

442 

30 

444 

444 

31 

458 

472 

32 

474 

979 

N 

13 

18 

23 

46 

53 

39 

17 

17 

IS 

15 

3 


TEST 

DET 

0 

2 

3 

7 

12 

12 

6 

1 

1 

1 

0 


7  Rl< 

Cl.lCLL  Gi. . 

'  M  ■ 

(26) 

95'’. 

0  MI 

Ss  1  M 1 

0 

0 

0 

M 

9 

0 

1 

11 

3 

0 

1  1 

14 

t 

0 

21 

13 

0 

29 

18 

0 

0 

32 

16 

0 

0 

3 

0 

0 

0 

3 

0 

0 

0 

4 

0 

0 

ri 

3 

1 

20 

1 

2 

1 

29 

2 

0 

0 

0 

0 

0 

0 

,  0 

0 

4 

4 

84 

4? 

1? 

16 

90 

75 

3 

3 

79 

36 

7 

5 

63 

34 

6 

2 

26 

6 

10 

5 

45 

22 

12 

4 

29 

12 

11 

11 

93 

i'  6 

4 

3 

61 

24 

5 

5 

87 

54 

2 

2 

70 

22 

1 

0 

0 

0 

1 

0 

0 

0 

1 

1 

50 

1 

1 

50 

5 

8 

8 

91 

68 

59 

54 

90 

S3 

0 

0 

0 

0 

0 

0 

29 

0 

n 

6 

0 

0 

IS 

n 

6 

u 

0 

0 

0 

4; 


0 

0 


ii 


U 

0 

44 

M 


0 


Measured  by  Operator  A.  Lab.  Env. 


(b)  Optimum  Probability  Method  of  Data  Cumulation 
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(c)  Overlapping  Sixty  Point  Method  of  Data  Cumulation 
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Figure  D-26  (Concluded) 


(a)  Range  Interval  Method  of  Data  Cumulation 
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(a)  Range  Interval  Method  of  Data  Cumulation 
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(b)  Optimum  Probability  Method  of  Data  Cumulation 
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Figure  D-28  (Contitiued) 
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23-JUL 

-75 

RADIOGRAPHY 

TEST  2, 

u 

(30).  ROCKWELL  SC 

RANGE 

NIN  LN 

MAK  LH 

IS  * 

N 

DET  ? 

50*. 

95'. 

0  MISS 

1  1 

1 

7 

■'  8 

0 

8 

0 

0 

2 

20 

.  ■■■  28  ' 

:l  11 

3 

0 

7 

0 

0 

3 

£0 

40 

26 

4 

0 

5 

0 

0 

4 

20 

51 

45 

8 

0 

9 

0 

0 

5 

52 

62 

26 

3 

0 

16 

0 

0 

6 

52 

73 

69 

21 

.0 

21 

0 

0 

"f 

1 

74 

84 

38 

15 

0 

26 

0 

0 

8 

74 

95 

64 

27 

0 

31 

0 

0 

9 

74 

106 

83 

33 

0 

30 

0 

0 

le 

74 

117 

91 

34 

0 

28 

0 

0 

11 

63 

126 

147 

49 

0 

26 

0 

0 

12 

52 

138 

189 

59 

0 

25 

0 

0 

13 

52 

150 

202 

61 

0 

24 

0 

0 

14 

52 

158 

208 

62 

0 

24 

0 

0 

15 

52 

171 

210 

63 

0 

24 

0 

0 

16 

52 

183 

212 

64 

0 

25 

0 

0 

17 

52 

190 

214 

65 

0 

25 

0 

0 

18 

63 

197 

189 

53 

0 

25 

0 

0 

19 

0 

0 

0 

0 

0 

0 

0 

0 

20 

0 

0 

0 

0 

0 

0 

0 

0 

21 

0 

0 

0 

0 

0 

0 

0 

0 

22 

197 

249 

8 

8 

0 

68 

0 

0 

23 

197 

260 

19 

19 

0 

85 

10 

27 

24 

197 

269 

24 

24 

0 

88 

5 

22 

25 

197 

279 

27 

27 

0 

89 

2 

19 

26 

197 

290 

32 

32 

0 

91 

0 

14 

27 

197 

304 

37 

36 

0 

87 

9 

24 

28 

197 

313 

44 

41 

0 

83 

32 

45 

29 

197 

326 

52 

44 

0 

1^3 

U 

0 

30 

197 

336 

60 

48 

0 

69 

0 

0 

31 

197 

347 

69 

54 

0 

68 

0 

0 

32 

197 

362 

73 

57 

0 

68 

0 

0 

Crack  Length  (Mil)* 

50  100  150  200  250  300  350  400 


Figure  0-30  (Continued) 


D-91 
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Figure  D-37  (Continued) 
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Figure  D-38  Probability  of  Detection  for  Ti-6A1-4V  Using 

Liquid  Penetrant.  Fatigue  Cracks  in  Flat  Plates. 
Prod.  Env.  D-118 
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Figure  D-38  (Continued) 
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(c)  Overlapping  Sixty  Point  Method  of  Data  Cumulation 
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Pigvice  D-38  (Concluded) 
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(a)  Range  Interval  Method  of  Data  Cumulation 
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Figure  D-39  Probability  of  Detection  for  Ti-6A1-4V  Using 

Liquid  Penetrant.  Fatigue  Cracks  in  Flat  Plates. 
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Figure  D-39  (Concluded) 
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(c)  Overlapping  Sixty  Point  Method  of  Data  Cumulation 
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(b)  Optimum  Probability  Method  of  Data  Cumulation 
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figure. D-63  (Continued) 


(a)  Range  Interval  Method  of  Data  Cumulation 
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(a)  Range  Interval  Method  of  Data  Cumulation 
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(d)  Range  Interval  Method  of  Data  Cumulation 
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(a)  Range  Interval  Method  of  Data  Cumulation 
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(b)  Optimum  Probability  Method  of  Data  Cumulation 
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(b)  Optimum  Probability  Method  of  Data  Cumulation 
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(c)  Overlapping  Sixty  Point  Method  of  Data  Cumulation 
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(a)  Range  Interval  Method  of  Data  Cumulation 
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(b)  Optimtim  Probability  Method  of  Data  Cumulation 


28-0CT-r5 
RANGE  MIN  LN 

1  30 

2  60- 

3  0 


4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 
21 
22 

23 

24 

25 

26 

27 

28 

29 

30 

31 

32 


60 

60 

60 

0 

60 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

60 


NAGMETIC 
LH 
.  50 

70 

0 

120 

140 

160 

0 

200 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

750 


FhP.TICLE 

31 

■  ■'  <27 

0 
52 
90 
103 
0 

106 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 
0 

112 


TEST 
C€T 
24 
27 
•  0 
52 
90 
103 
0 

106 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

A 

112 


SO'i 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 


Crack  Length  (Mils)* 


.•  (98) 

95?;  0  MISS 

61  0 

89  2 

0  0 

94  0 

96  0 

97  0 

0  0 

97  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

0  0 

A  a 

97  0 


i  MISS 
1 


Figure  b-98  (Continued) 


D-301 


O  O  O  O  O  O  C3  O  Q  O  O  O  O  O  O  O  O  O  <S>  O  O  *3  O  O  O  O 


(c)  Overlapping  Sixty  Point  Method  of  Data  Cumulation 


Figure  iy-98  '  (Concluded) _ ' ZI-Z* 

n-?02 


Flgute 


(a)  Range  Interval  Method  of  Data  Cumulation 


IS-OCT 

-75 

MrtGNETIC  F'NRTICLE 

TEST 

1 

’> 

(99) 

iNGE 

MIN  LN 

j.  MAX  .LN  . 

N 

DET 

5ir; 

95X 

D  MI5S 

J  flISS 

1 

70' 

20 

20 

86 

9 

2^: 

2 

90; 

90  1 . 

■M9 

19 

96 

85 

10 

27 

3 

0 

0 

0 

0. 

0 

0 

0 

0 

4 

110 

110 

8 

6 

67 

40 

0 

0 

5 

120 

130 

12 

12 

94 

77 

17 

34 

6 

140 

140 

16 

16 

95 

82 

13 

30 

7 

0 

0 

0 

0 

0 

0 

0 

0 

8 

160 

160 

2 

2 

70 

22 

0 

0 

9 

0 

0 

0 

0 

0 

0 

0 

0 

10 

180 

180 

9 

9 

92 

71 

0 

0 

11 

0 

0 

0 

0 

0 

0 

0 

0 

12 

0 

0 

0 

0 

0 

0 

0 

0 

13 

0 

0 

0 

0 

0 

0 

0 

0 

14 

0 

0 

0 

0 

0 

0 

0 

0 

15 

250 

250 

1 

1 

50 

5 

0 

0 

16 

0 

0 

0 

0 

0 

0 

0 

0 

17 

270 

270 

2 

2 

70 

22 

0 

0 

18 

280 

280 

1 

1 

50 

5 

0 

0 

19 

300 

300 

11 

11 

93 

76 

18 

35 

20 

310 

310 

1 

1 

50 

5 

0 

0 

21 

0 

0 

0 

0 

0 

0 

0 

0 

22 

0 

0 

0 

0 

0 

0 

0 

0 

23 

0 

0 

0 

0 

0 

0 

0 

0 

24 

0 

0 

0 

0 

0 

0 

0 

0 

25 

0 

0 

0 

0 

0 

0 

0 

0 

26 

0 

0 

0 

0 

0 

0 

0 

0 

27 

390 

390 

1 

1 

50 

5 

6 

0 

28 

0 

0 

0 

0 

0 

0 

0 

0 

29 

0 

0 

0 

0 

0 

0 

ij 

0 

30 

0 

0 

0 

0 

0 

0 

n 

c 

31  ■ 

0 

e 

0 

n 

0 

0 

0 

0 

32 

450 

460 

3 

3 

79 

36 

0 

0 

D-99  Probability  of  Detection  for  4340M  Steel  Using  Magnetic  Particles 
Compressed  Notch  Flaws  in  Hollow  Filleted  Cylinder.  Lab.  Env. 

D-303 


Lower  Confidence  Limit 


(b)  Optimum  Probability  Method  of  Data  Cumulation 
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(c)  Overlapping  Sixty  Point  Method  of  Data  Cumulation 
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(a)  Range  Interval  Method  of  Data  Cumulation 
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(c)  Overlapping  Sixty  Point  Method  of  Data  Cumulation 
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(b)  Optimum  Probability  Method  of  Data  Cumulation 
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(c)  Overlapping  Sixty  Point  Method  of  Data  Cumulation 


(a)  Range  Interval  Method  of  Data  Cumulation 
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(a)  Range  Interval  Method  of  Data  Cumulation 
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(c)  Overlapping  Sixty  Point  Method  of  Data  Cumulation 
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(a)  Range  Interval  Method  of  Data  Cumulation 
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(b)  Optimum  Probability  Method  of  Data  Cumulation 
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